Importing the Datasets¶
In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
In [2]:
hosp = pd.read_csv('Hospitalisation details.csv')
medi = pd.read_csv('Medical Examinations.csv')
name = pd.read_excel('Names.xlsx')
In [3]:
hosp.info()
In [4]:
medi.info()
In [5]:
name.info()
1. Collate the files so that all the information is in one place¶
In [6]:
table1 = pd.merge(hosp,medi, on = "Customer ID")
In [7]:
dataset = pd.merge(table1,name, on = "Customer ID")
In [8]:
dataset
Out[8]:
2. Check for missing values in the dataset¶
In [9]:
dataset.isnull().sum()
Out[9]:
In [ ]:
In [ ]:
3. Find the percentage of rows that have trivial value (for example, ?), and delete such rows if they do not contain significant information¶
In [10]:
trivial_rows = dataset[dataset == "?"].count(axis=1).sum()
In [11]:
trivial_rows
Out[11]:
In [12]:
total_rows = dataset.shape[0]
total_rows
Out[12]:
In [13]:
percentage = (trivial_rows / total_rows) * 100
In [14]:
percentage
Out[14]:
In [15]:
print("Percentage of trivial rows: {:.2f}%".format(percentage))
In [16]:
dataset = dataset[dataset != "?"].dropna()
In [17]:
dataset.shape
Out[17]:
4. Use the necessary transformation methods to deal with the nominal and ordinal categorical variables in the dataset¶
In [18]:
dataset_cat = dataset.select_dtypes(exclude='number')
In [19]:
dataset_cat.columns
Out[19]:
In [20]:
dataset['Heart Issues'].value_counts()
Out[20]:
In [21]:
dataset['Any Transplants'].value_counts()
Out[21]:
In [22]:
from sklearn.preprocessing import LabelEncoder
le= LabelEncoder()
In [23]:
dataset["Heart Issues"]=le.fit_transform(dataset["Heart Issues"])
dataset["Any Transplants"]=le.fit_transform(dataset["Any Transplants"])
dataset["Cancer history"]=le.fit_transform(dataset["Cancer history"])
dataset["smoker"]=le.fit_transform(dataset["smoker"])
In [24]:
dataset["Heart Issues"].value_counts()
Out[24]:
In [25]:
dataset['Hospital tier'].value_counts()
Out[25]:
In [26]:
dataset['Hospital tier'] = dataset['Hospital tier'].str.replace('tier', '')
#df['col1'] = df['col1'].str.replace('example-', '')
In [27]:
dataset['Hospital tier'].value_counts()
Out[27]:
In [28]:
dataset['Hospital tier'] = dataset['Hospital tier'].str.replace("-", "")
dataset['Hospital tier'] = dataset['Hospital tier'].astype(int)
In [29]:
dataset
Out[29]:
In [30]:
dataset['City tier'] = dataset['City tier'].str.replace("tier", "")
dataset['City tier'] = dataset['City tier'].str.replace("-", "")
In [31]:
dataset
Out[31]:
In [32]:
dataset['City tier'] = dataset['City tier'].astype(int)
In [33]:
dataset
Out[33]:
5. The dataset has State ID, which has around 16 states. All states are not represented in equal proportions in the data. Creating dummy variables for all regions may also result in too many insignificant predictors. Nevertheless, only R1011, R1012, and R1013 are worth investigating further. Create a suitable strategy to create dummy variables with these restraints.¶
In [34]:
dataset['State ID'].value_counts()
Out[34]:
In [35]:
dataset['state_group'] = np.where((dataset['State ID'] == 'R1011') | (dataset['State ID'] == 'R1012') | (dataset['State ID'] == 'R1013'), dataset['State ID'], 'other')
In [36]:
state_dummies = pd.get_dummies(dataset['state_group'], prefix='state')
df = pd.concat([dataset, state_dummies], axis=1)
In [37]:
dataset
Out[37]:
In [38]:
dataset=dataset[dataset["State ID"].isin(['R1011','R1012','R1013'])]
dataset.shape
dataset["State ID"]=le.fit_transform(dataset["State ID"])
dataset["State ID"].unique()
Out[38]:
In [39]:
dataset['state_group'].value_counts()
Out[39]:
In [ ]:
In [40]:
dataset["state_group"].replace('R1011',1,inplace=True)
dataset["state_group"].replace('R1012',2,inplace=True)
dataset["state_group"].replace('R1013',3,inplace=True)
dataset["state_group"].replace('other',0,inplace=True)
In [41]:
dataset
Out[41]:
6. The variable NumberOfMajorSurgeries also appears to have string values. Apply a suitable method to clean up this variable.¶
In [42]:
dataset["NumberOfMajorSurgeries"].replace('No major surgery',0,inplace=True)
In [43]:
dataset
Out[43]:
7. Age appears to be a significant factor in this analysis. Calculate the patients' ages based on their dates of birth.¶
In [44]:
month_dict = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6, 'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}
In [45]:
dataset['month'] = dataset['month'].map(month_dict)
In [46]:
dataset
Out[46]:
In [47]:
dataset.year = dataset.year.astype(int)
In [48]:
dataset['age'] = 2023 - dataset.year
In [49]:
dataset
Out[49]:
In [ ]:
8. The gender of the patient may be an important factor in determining the cost of hospitalization. The salutations in a beneficiary's name can be used to determine their gender. Make a new field for the beneficiary's gender.¶
In [50]:
gender= ['0' if 'Mr.' in name else '1' for name in dataset['name']]
dataset["Gender"]=gender
dataset.head()
Out[50]:
9. You should also visualize the distribution of costs using a histogram, box and whisker plot, and swarm plot.¶
In [51]:
plt.figure(figsize=(15,8))
sns.histplot(dataset['charges'])
plt.title('Distribution of cost')
plt.show()
In [52]:
plt.figure(figsize=(15,8))
sns.boxplot(dataset['charges'])
plt.title('Distribution of cost')
plt.show()
In [53]:
plt.figure(figsize=(15,8))
sns.swarmplot(dataset['charges'])
plt.title('Distribution of cost')
plt.show()
10. State how the distribution is different across gender and tiers of hospitals¶
In [54]:
plt.figure(figsize=(15,8))
sns.boxplot(x = 'charges', y = 'Gender',data = dataset)
plt.title('Distribution of cost')
plt.show()
In [55]:
plt.figure(figsize = (15,5))
sns.boxplot(x = "City tier",y = "charges", data = dataset)
plt.show()
11. Create a radar chart to showcase the median hospitalization cost for each tier of hospitals¶
In [56]:
median = dataset.groupby('Hospital tier')[['charges']].median().reset_index()
In [58]:
import plotly.express as px
fig = px.line_polar(median, r='charges', theta='Hospital tier', line_close=True)
fig.show()
12. Create a frequency table and a stacked bar chart to visualize the count of people in the different tiers of cities and hospitals¶
In [59]:
table = pd.crosstab(dataset['City tier'], dataset['Hospital tier'])
print(table)
In [60]:
table.plot(kind='bar', stacked=True)
plt.xlabel('City tier')
plt.ylabel('Hospital tier')
plt.title('Count of People in Different Tiers of Cities and Hospitals')
plt.show()
In [61]:
import scipy.stats as stats
print('Null Hypothesis => Average hospitalization costs for the three types of hospitals are not significantly different.')
f_val, p_val = stats.f_oneway(dataset[dataset['Hospital tier'] == 'tier,1']['charges'],
dataset[dataset['Hospital tier'] == 'tier,2']['charges'],
dataset[dataset['Hospital tier'] == 'tier,3']['charges'])
print('P-value :',p_val)
if p_val < 0.05:
print("Reject null hypothesis")
else:
print("Accept null hypothesis")
b. The average hospitalization costs for the three types of cities are not significantly different¶
In [62]:
print('Null Hypothesis => Average hospitalization costs for the three types of cities are not significantly different.')
f_val, p_val = stats.f_oneway(dataset[dataset['City tier'] == 'tier,1']['charges'],
dataset[dataset['City tier'] == 'tier,2']['charges'],
dataset[dataset['City tier'] == 'tier,3']['charges'])
print('P-value :',p_val)
if p_val < 0.05:
print("Reject null hypothesis")
else:
print("Accept null hypothesis")
c. The average hospitalization cost for smokers is not significantly different from the average cost for nonsmokers¶
In [63]:
print('Null Hypothesis => Average hospitalization costs for smokers is not significantly different from the average cost for nonsmokers.')
t_val, p_val = stats.ttest_ind(dataset[dataset['smoker'] == 'yes']['charges'],
dataset[dataset['smoker'] == 'no']['charges'])
print('P-value :',p_val)
if p_val < 0.05:
print("Reject null hypothesis")
else:
print("Accept null hypothesis")
d. Smoking and heart issues are independent¶
In [64]:
from scipy.stats import chi2_contingency
contingency_table = pd.crosstab(dataset['smoker'], dataset['Heart Issues'])
chi2, p, dof, expected = chi2_contingency(contingency_table)
print(f'P-value = {p}')
if p < 0.05:
print("Reject the null hypothesis, Smoking and heart issues are independent.")
else:
print("Accept null hypothesis, Smoking and heart issues are independent.")
2. Machine Learning¶
1. Examine the correlation between predictors to identify highly correlated predictors. Use a heatmap to visualize this.¶
In [65]:
dataset.drop(["Customer ID",'name'], inplace=True, axis=1)
In [66]:
plt.figure(figsize=(15,10))
sns.heatmap(dataset.corr(),square=True,annot=True,linewidths=1)
Out[66]:
2. Develop and evaluate the final model using regression with a stochastic gradient descent optimizer. Also, ensure that you apply all the following suggestions:¶
• Perform the stratified 5-fold cross-validation technique for model building and validation • Use standardization and hyperparameter tuning effectively • Use sklearn-pipelines • Use appropriate regularization techniques to address the bias-variance trade-off
a. Create five folds in the data, and introduce a variable to identify the folds¶
b. For each fold, run a for loop and ensure that 80 percent of the data is used to train the model and the remaining 20 percent is used to validate it in each iteration¶
c. Develop five distinct models and five distinct validation scores (root mean squared error values)¶
d. Determine the variable importance scores, and identify the redundant variables¶
In [67]:
from sklearn.model_selection import train_test_split
In [68]:
x = dataset.drop(["charges"], axis=1)
y = dataset['charges']
x_train, x_test, y_train, y_test = train_test_split(x,y, test_size=.20,random_state=10)
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
x_train = sc.fit_transform(x_train)
x_test = sc.fit_transform(x_test)
from sklearn.linear_model import SGDRegressor
In [69]:
from sklearn.model_selection import GridSearchCV
params = {'alpha': [0.0001, 0.001, 0.01, 0.05, 0.1, 0.2,0.3,0.4,0.5,
0.6,0.7,0.8,0.9,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,
9.0,10.0,20,50,100,500,1000],
'penalty': ['l2', 'l1', 'elasticnet']}
sgd = SGDRegressor()
# Cross Validation
folds = 5
model_cv = GridSearchCV(estimator = sgd,
param_grid = params,
scoring = 'neg_mean_absolute_error',
cv = folds,
return_train_score = True,
verbose = 1)
model_cv.fit(x_train,y_train)
Out[69]:
In [70]:
model_cv.best_params_
Out[70]:
In [71]:
sgd = SGDRegressor(alpha= 100, penalty= 'l1')
In [72]:
sgd.fit(x_train, y_train)
Out[72]:
In [73]:
sgd.score(x_test, y_test)
Out[73]:
In [74]:
y_pred = sgd.predict(x_test)
In [75]:
from sklearn.metrics import mean_squared_error, mean_absolute_error
sgd_mae = mean_absolute_error(y_test, y_pred)
sgd_mse = mean_squared_error(y_test, y_pred)
sgd_rmse = sgd_mse*(1/2.0)
In [76]:
print("MAE:", sgd_mae)
print("MSE:", sgd_mse)
print("RMSE:", sgd_rmse)
In [77]:
importance = sgd.coef_
pd.DataFrame(importance, index = x.columns, columns=['Feature_imp'])
Out[77]:
3. Use random forest and extreme gradient boosting for cost prediction, share your crossvalidation results, and calculate the variable importance scores
1.0.3 random forest¶
In [78]:
from sklearn.ensemble import RandomForestRegressor
In [79]:
rf = RandomForestRegressor(n_estimators = 1000, random_state = 42)
rf.fit(x_train, y_train)
Out[79]:
In [80]:
score = rf.score(x_test,y_test)
score
Out[80]:
In [81]:
y_pred = rf.predict(x_test)
rf_mae = mean_absolute_error(y_test, y_pred)
rf_mae
Out[81]:
1.0.4 extreme gradient boosting¶
In [82]:
from sklearn.ensemble import GradientBoostingRegressor
In [83]:
gbr = GradientBoostingRegressor(n_estimators = 1000, random_state = 42)
gbr.fit(x_train, y_train)
Out[83]:
In [84]:
score = gbr.score(x_test,y_test)
score
Out[84]:
In [85]:
y_pred = gbr.predict(x_test)
gbr_mae = mean_absolute_error(y_test, y_pred)
gbr_mae
Out[85]:
- Case scenario: Estimate the cost of hospitalization for Christopher, Ms. Jayna (her date of birth is 12/28/1988, height is 170 cm, and weight is 85 kgs). She lives in a tier-1 city and her state’s State ID is R1011. She lives with her partner and two children. She was found to be nondiabetic (HbA1c = 5.8). She smokes but is otherwise healthy. She has had no transplants or major surgeries. Her father died of lung cancer. Hospitalization costs will be estimated using tier-1 hospitals.
In [107]:
dataset.columns
Out[107]:
In [108]:
df= pd.DataFrame({ 'year' : [1998], 'month' : [12] ,'date': [28],
'city_tier' : [1], 'children' :[ 2],
'HbA1c' : [5.8],
'smoker_yes' : [1],
'heart_issues_yes' : [0],
'any_transplants_yes' : [0],
'numberofmajorsurgeries' :[ 0],
'cancer_history_yes' : [1],
'hospital_tier' : [1],
'bmi' : [85/(1.70 **2)], 'age' : [25],'Gender' : [0],'state_group' : [1],
'state_id_R1011' : [1]
})
In [102]:
df
Out[102]:
5. Find the predicted hospitalization cost using all five models. The predicted value should be the mean of the five models' predicted values'.¶
In [92]:
Hospital_cost = []
In [109]:
Cost1 = sgd.predict(df)
Hospital_cost.append(Cost1)
Cost2 = rf.predict(df)
Hospital_cost.append(Cost2)
Cost3 = gbr.predict(df)
Hospital_cost.append(Cost3)
avg_cost = np.mean(Hospital_cost)
avg_cost
Out[109]:
In [ ]: